Updates for Rivet 4 #44603

mseidel42 · 2024-04-03T16:12:28Z

PR description:

Updates for Rivet 4, needs cms-sw/cmsdist#9111

Rivet 4 interface changes
Moved to HepMC3 (strict requirement from Rivet now)
Fixed problem in MergedGenParticleProducer (which caused HepMC3 to crash)
Added more unit tests

PR validation:

% scram b -j 8 runtests
>> Local Products Rules ..... started
>> Local Products Rules ..... done
>> Creating project symlinks
Creating test log file logs/el8_amd64_gcc12/testing.log
Pass   11s ... GeneratorInterface/RivetInterface/test-particleLevel_fromHepMC3
Pass   12s ... GeneratorInterface/RivetInterface/test-HTXS
Pass   15s ... GeneratorInterface/RivetInterface/test-particleLevel_fromGenParticles
Pass   18s ... GeneratorInterface/RivetInterface/test-genParticles2HepMC
Pass   19s ... GeneratorInterface/RivetInterface/test-particleLevel_fromMiniAod
Pass   19s ... GeneratorInterface/RivetInterface/test-rivet-run
Pass    2s ... GeneratorInterface/RivetInterface/test-yoda-merge
Pass    3s ... GeneratorInterface/RivetInterface/test-yoda-root
Pass   29s ... GeneratorInterface/RivetInterface/test-rivet-list
Pass   14s ... GeneratorInterface/RivetInterface/test-rivet-plot

cmsbuild · 2024-04-03T16:12:56Z

cms-bot internal usage

cmsbuild · 2024-04-03T16:18:21Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44603/39791

This PR adds an extra 52KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File SimDataFormats/GeneratorProducts/src/classes.h modified in PR(s): Adding HepMC3 products to EDMProduct #38033
- File SimDataFormats/GeneratorProducts/src/classes_def.xml modified in PR(s): Adding HepMC3 products to EDMProduct #38033

cmsbuild · 2024-04-03T16:18:48Z

A new Pull Request was created by @mseidel42 for master.

It involves the following packages:

GeneratorInterface/Core (generators)
GeneratorInterface/RivetInterface (generators)
SimDataFormats/GeneratorProducts (generators)

@GurpreetSinghChahal, @SiewYan, @cmsbuild, @mkirsano, @alberto-sanchez, @bbilin, @menglu21 can you please review it and eventually sign? Thanks.
@fabiocos, @mkirsano, @apsallid, @youyingli, @rovere, @alberto-sanchez, @bsunanda, @missirol this is something you requested to watch as well.
@sextonkennedy, @antoniovilela, @rappoccio you are the release manager for this.

cms-bot commands are listed here

iarspider · 2024-04-03T17:01:07Z

please test

iarspider · 2024-04-03T17:02:56Z

please test with cms-sw/cmsdist#9111

cmsbuild · 2024-04-03T19:47:20Z

-1

Failed Tests: UnitTests RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8f5d71/38581/summary.html
COMMIT: 0d18220
CMSSW: CMSSW_14_1_X_2024-04-03-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44603/38581/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8f5d71/38581/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8f5d71/38581/git-merge-result

Unit Tests

I found 2 errors in the following unit tests:

---> test test-genParticles2HepMC had ERRORS
---> test test-rivet-plot had ERRORS

RelVals

135.4

----- Begin Fatal Exception 03-Apr-2024 20:32:03 CEST-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 2 stream: 1
   [1] Running path 'NANOEDMAODSIMoutput_step'
   [2] Prefetching for module PoolOutputModule/'NANOEDMAODSIMoutput'
   [3] Calling method for module SimpleGenEventFlatTableProducer/'genTable'
   Additional Info:
      [a] Fatal Root Error: @SUB=TClass::LoadClassInfo
no interpreter information for class GenEventInfoProduct is available even though it has a TClass initialization routine.

----- End Fatal Exception -------------------------------------------------

1330.0

----- Begin Fatal Exception 03-Apr-2024 20:44:11 CEST-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'NANOEDMAODSIMoutput_step'
   [2] Prefetching for module PoolOutputModule/'NANOEDMAODSIMoutput'
   [3] Calling method for module SimpleGenEventFlatTableProducer/'genTable'
   Additional Info:
      [a] Fatal Root Error: @SUB=TClass::LoadClassInfo
no interpreter information for class GenEventInfoProduct is available even though it has a TClass initialization routine.

----- End Fatal Exception -------------------------------------------------

11634.0

----- Begin Fatal Exception 03-Apr-2024 20:51:30 CEST-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'dqmoffline_2_step'
   [2] Prefetching for module NanoAODDQM/'nanoDQMMC'
   [3] Calling method for module SimpleGenEventFlatTableProducer/'genTable'
   Additional Info:
      [a] Fatal Root Error: @SUB=TInterpreter::AutoParse
Error parsing payload code for class GenEventInfoProduct with content:

#line 1 "SimDataFormatsGeneratorProducts_xr dictionary payload"

#ifndef CMS_DICT_IMPL
  #define CMS_DICT_IMPL 1
#endif
#ifndef _REENTRANT
  #define _REENTRANT 1
#endif
#ifndef GNUSOURCE
  #define GNUSOURCE 1
#endif
#ifndef __STRICT_ANSI__
  #define __STRICT_ANSI__ 1
#endif
#ifndef GNU_GCC
  #define GNU_GCC 1
#endif
#ifndef _GNU_SOURCE
  #define _GNU_SOURCE 1
#endif
#ifndef TBB_USE_GLIBCXX_VERSION
  #define TBB_USE_GLIBCXX_VERSION 120301
#endif
#ifndef TBB_SUPPRESS_DEPRECATED_MESSAGES
  #define TBB_SUPPRESS_DEPRECATED_MESSAGES 1
#endif
#ifndef TBB_PREVIEW_RESUMABLE_TASKS
  #define TBB_PREVIEW_RESUMABLE_TASKS 1
#endif
#ifndef TBB_PREVIEW_TASK_GROUP_EXTENSIONS
  #define TBB_PREVIEW_TASK_GROUP_EXTENSIONS 1
#endif
#ifndef BOOST_SPIRIT_THREADSAFE
  #define BOOST_SPIRIT_THREADSAFE 1
#endif
#ifndef PHOENIX_THREADSAFE
  #define PHOENIX_THREADSAFE 1
#endif
#ifndef BOOST_MATH_DISABLE_STD_FPCLASSIFY
  #define BOOST_MATH_DISABLE_STD_FPCLASSIFY 1
#endif
#ifndef BOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX
  #define BOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX 1
#endif
#ifndef CMSSW_GIT_HASH
  #define CMSSW_GIT_HASH "CMSSW_14_1_X_2024-04-03-1100"
#endif
#ifndef PROJECT_NAME
  #define PROJECT_NAME "CMSSW"
#endif
#ifndef PROJECT_VERSION
  #define PROJECT_VERSION "CMSSW_14_1_X_2024-04-03-1100"
#endif
#ifndef CMSSW_REFLEX_DICT
  #define CMSSW_REFLEX_DICT 1
#endif

#define _BACKWARD_BACKWARD_WARNING_H
// Inline headers
#include <utility>
#include <vector>
#include <map>
#include <set>

#include "DataFormats/Common/interface/Wrapper.h"
#include "DataFormats/Common/interface/RefVector.h"

#include "SimDataFormats/GeneratorProducts/interface/LHEEventProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/LHERunInfoProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/LHEXMLStringProduct.h"

#include "SimDataFormats/GeneratorProducts/interface/HepMCProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/HepMC3Product.h"

#include "SimDataFormats/GeneratorProducts/interface/GenRunInfoProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/GenFilterInfo.h"
#include "SimDataFormats/GeneratorProducts/interface/GenEventInfoProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/GenEventInfoProduct3.h"
#include "SimDataFormats/GeneratorProducts/interface/GenLumiInfoProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/GenLumiInfoHeader.h"
#include "SimDataFormats/GeneratorProducts/interface/ExternalGeneratorLumiInfo.h"
#include "SimDataFormats/GeneratorProducts/interface/ExternalGeneratorEventInfo.h"

#include <HepMC/GenRanges.h>

//needed for backward compatibility between HepMC 2.06.xx and 2.05.yy
namespace hepmc_rootio {
  void add_to_particles_in(HepMC::GenVertex*, HepMC::GenParticle*);
  void clear_particles_in(HepMC::GenVertex*);

  inline void weightcontainer_set_default_names(unsigned int n,
                                                std::map<std::string, HepMC::WeightContainer::size_type>& names) {
    std::ostringstream name;
    for (HepMC::WeightContainer::size_type count = 0; count < n; ++count) {
      name.str(std::string());
      name << count;
      names[name.str()] = count;
    }
  }
}  // namespace hepmc_rootio

#undef  _BACKWARD_BACKWARD_WARNING_H


----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

13234.013234.0_TTbar_14TeV+2021FS/step2_TTbar_14TeV+2021FS.log
14034.014034.0_TTbar_14TeV+2023FS/step2_TTbar_14TeV+2023FS.log
13434.013434.0_TTbar_14TeV+2021FSPU/step2_TTbar_14TeV+2021FSPU.log

Expand to see more relval errors ...

mseidel42 · 2024-04-04T08:44:12Z

Hm, I don't understand these two:

Unit Tests

---> test test-genParticles2HepMC had ERRORS

Actually fails on exit, sometimes yes, sometimes not:

T---Report end!

Fatal system signal has occurred during exit
timeout: the monitored command dumped core
/bin/sh: line 1: 3628348 Aborted                 timeout 3600 sh -c 'cmsRun /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-04-03-1100/src/GeneratorInterface/RivetInterface/test/genParticles2HepMC_cfg.py'

---> test test-genParticles2HepMC had ERRORS
TestTime:61
^^^^ End Test test-genParticles2HepMC ^^^^

Could this be related to #40347? It occurs also when I run just the MergedGenParticleProducer multiple times in a row, and that produces nothing but a reco::GenParticleCollection, same as the usual GenParticleProducer, so no additional dictionary should be needed?

RelVals

Another transient dictionary problem? It passes in my local area 😕

% runTheMatrix.py -l 135.4
...
Running up to 4 concurrent jobs, each with 1 thread per process

Preparing to run 135.4 ZEEFS_13

...
 
135.4_ZEEFS_13 Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Thu Apr  4 10:31:43 2024-date Thu Apr  4 10:25:50 2024; exit: 0 0 0 0
1 1 1 1 tests passed, 0 0 0 0 failed

cmsbuild · 2024-04-04T08:53:01Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44603/39795

This PR adds an extra 56KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File SimDataFormats/GeneratorProducts/src/classes.h modified in PR(s): Adding HepMC3 products to EDMProduct #38033
- File SimDataFormats/GeneratorProducts/src/classes_def.xml modified in PR(s): Adding HepMC3 products to EDMProduct #38033

cmsbuild · 2024-04-04T08:53:29Z

Pull request #44603 was updated. @mkirsano, @GurpreetSinghChahal, @menglu21, @alberto-sanchez, @SiewYan, @bbilin, @cmsbuild can you please check and sign again.

menglu21 · 2024-07-02T07:48:16Z

+1

cmsbuild · 2024-07-02T07:48:45Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @sextonkennedy, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)
Notice This PR was tested with additional Pull Request(s), please also merge them if necessary: cms-sw/cmsdist#9111

rappoccio · 2024-07-02T15:32:59Z

+1

perrotta · 2024-07-03T06:02:15Z

Quite several workflows are failing since this PR was merged in the CMSSW_14_1_X_2024-07-02-2300 IB with error like:

----- Begin Fatal Exception 03-Jul-2024 07:06:10 CEST-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 8 stream: 1
   [1] Running path 'NANOEDMAODSIMoutput_step'
   [2] Prefetching for module PoolOutputModule/'NANOEDMAODSIMoutput'
   [3] Prefetching for module SimpleHTXSFlatTableProducer/'HTXSCategoryTable'
   [4] Prefetching for module HTXSRivetProducer/'rivetProducerHTXS'
   [5] Calling method for module GenParticles2HepMCConverter/'genParticles2HepMCHiggsVtx'
Exception Message:
A std::exception was thrown.
GenCrossSection::xsec(const unsigned long&): index outside of range
----- End Fatal Exception -------------------------------------------------

Other PRs were merged at the same time, but I have the impression that this PR could be the culprit: @mseidel42 could you please have a look?

mseidel42 · 2024-07-03T06:05:06Z

Hi @perrotta, can you give me an example workflow number for testing, please?

smuzaffar · 2024-07-03T06:30:30Z

@mseidel42 workflow number 11601.0 step3 (see https://cmssdt.cern.ch/SDT/html/cmssdt-ib/#/relVal/CMSSW_14_1/2024-07-02-2300?selectedArchs=el8_amd64_gcc12&selectedFlavors=X&selectedStatus=failed ) is failing with this error. You can run runTheMAtrix.py -l 11601.0 --ibeos -t 4 to reproduce it

GeneratorInterface/RivetInterface/plugins/GenParticles2HepMCConverter.cc

mseidel42 · 2024-07-04T05:53:00Z

Fixed merged: #45367

Let me know in case there are any problems with the morning IB 🙈

iarspider · 2024-07-05T13:10:47Z

@mseidel42 RelVals 546.0, 547.0 are failing with ProductNotFound, e.g.:

----- Begin Fatal Exception 03-Jul-2024 03:26:26 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 4 stream: 1
   [1] Running path 'dqmoffline_step'
   [2] Prefetching for module NanoAODDQM/'nanogenDQM'
   [3] Prefetching for module SimpleCandidateFlatTableProducer/'rivetLeptonTable'
   [4] Calling method for module ParticleLevelProducer/'particleLevel'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: edm::HepMC3Product
Looking for module label: generatorSmeared
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

mseidel42 · 2024-07-05T14:40:06Z

Ah, that must come from the fact that we cannot use the HepMC2 generator output directly anymore.

Instead we need HepMC2 -> GenParticles -> HepMC3, I will make the patch to the NanoGen configuration for that.

As this is the second thing that was not caught by the PR check before, maybe one ParticleGun and one NanoGen workflow could be added to the list of checks?

iarspider · 2024-07-05T15:17:07Z

@mseidel42 Also, only in el9 IBs, there is a "fatal system signal during exit" in test-particleLevel_fromMiniAod

mseidel42 · 2024-07-09T13:57:21Z

@mseidel42 Also, only in el9 IBs, there is a "fatal system signal during exit" in test-particleLevel_fromMiniAod

Do you have any hint how to tackle that issue?

The only related thread I could find is #40347 but I don't see from there how I should improve the code.

makortel · 2024-07-09T14:01:18Z

@mseidel42 Also, only in el9 IBs, there is a "fatal system signal during exit" in test-particleLevel_fromMiniAod

Do you have any hint how to tackle that issue?

Running the job in gdb and printing the stack trace on the point of the signal (gdb should automatically break there without any explicit breakpoints) has usually helped forward.

makortel · 2024-07-22T13:29:17Z

@mseidel42 Also, only in el9 IBs, there is a "fatal system signal during exit" in test-particleLevel_fromMiniAod

Now being followed up in #45521

cmsbuild added this to the CMSSW_14_1_X milestone Apr 3, 2024

mseidel42 mentioned this pull request Apr 3, 2024

Rivet 4.0.0 and YODA 2.0.0 cms-sw/cmsdist#9111

Merged

cmsbuild added generators-pending pending-signatures tests-pending orp-pending code-checks-pending labels Apr 3, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 3, 2024

cmsbuild added tests-started and removed tests-pending labels Apr 3, 2024

cmsbuild added the requires-external label Apr 3, 2024

cmsbuild added tests-rejected and removed tests-started labels Apr 3, 2024

cmsbuild added tests-pending code-checks-pending and removed tests-rejected requires-external code-checks-approved labels Apr 4, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 4, 2024

cmsbuild added tests-approved and removed tests-started labels Jun 24, 2024

cmsbuild added generators-approved fully-signed and removed generators-pending pending-signatures labels Jul 2, 2024

cmsbuild added orp-approved and removed orp-pending labels Jul 2, 2024

cmsbuild merged commit d191977 into cms-sw:master Jul 2, 2024
12 checks passed

smuzaffar reviewed Jul 3, 2024

View reviewed changes

GeneratorInterface/RivetInterface/plugins/GenParticles2HepMCConverter.cc Show resolved Hide resolved

mseidel42 mentioned this pull request Jul 3, 2024

Fix HepMC3 conversion on MC without event weights (particle guns) #45367

Merged

cmsbuild mentioned this pull request Jul 3, 2024

TICLv5 : Superclustering DNN #45333

Merged

makortel mentioned this pull request Jul 9, 2024

Avoid releasing WaitingTask too early in case of exception in WaitingThreadPool #45402

Merged

mseidel42 mentioned this pull request Jul 9, 2024

Fix NanoGen with Rivet4 #45411

Merged

aandvalenzuela mentioned this pull request Jul 22, 2024

Unit test test-particleLevel_fromMiniAod failing with fatal signal at exit #45521

Open

cmsbuild added the changes-dataformats label Jul 22, 2024

bbilin mentioned this pull request Sep 3, 2024

Adding HepMC3 products to EDMProduct #38033

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updates for Rivet 4 #44603

Updates for Rivet 4 #44603

mseidel42 commented Apr 3, 2024

cmsbuild commented Apr 3, 2024 •

edited

Loading

cmsbuild commented Apr 3, 2024

cmsbuild commented Apr 3, 2024

iarspider commented Apr 3, 2024 via email

iarspider commented Apr 3, 2024 via email

cmsbuild commented Apr 3, 2024

mseidel42 commented Apr 4, 2024

cmsbuild commented Apr 4, 2024

cmsbuild commented Apr 4, 2024

menglu21 commented Jul 2, 2024

cmsbuild commented Jul 2, 2024

rappoccio commented Jul 2, 2024

perrotta commented Jul 3, 2024

mseidel42 commented Jul 3, 2024

smuzaffar commented Jul 3, 2024

mseidel42 commented Jul 4, 2024

iarspider commented Jul 5, 2024

mseidel42 commented Jul 5, 2024

iarspider commented Jul 5, 2024 •

edited

Loading

mseidel42 commented Jul 9, 2024

makortel commented Jul 9, 2024

makortel commented Jul 22, 2024

Updates for Rivet 4 #44603

Updates for Rivet 4 #44603

Conversation

mseidel42 commented Apr 3, 2024

PR description:

PR validation:

cmsbuild commented Apr 3, 2024 • edited Loading

cmsbuild commented Apr 3, 2024

cmsbuild commented Apr 3, 2024

iarspider commented Apr 3, 2024 via email

iarspider commented Apr 3, 2024 via email

cmsbuild commented Apr 3, 2024

Unit Tests

RelVals

RelVals-INPUT

mseidel42 commented Apr 4, 2024

Unit Tests

RelVals

cmsbuild commented Apr 4, 2024

cmsbuild commented Apr 4, 2024

menglu21 commented Jul 2, 2024

cmsbuild commented Jul 2, 2024

rappoccio commented Jul 2, 2024

perrotta commented Jul 3, 2024

mseidel42 commented Jul 3, 2024

smuzaffar commented Jul 3, 2024

mseidel42 commented Jul 4, 2024

iarspider commented Jul 5, 2024

mseidel42 commented Jul 5, 2024

iarspider commented Jul 5, 2024 • edited Loading

mseidel42 commented Jul 9, 2024

makortel commented Jul 9, 2024

makortel commented Jul 22, 2024

cmsbuild commented Apr 3, 2024 •

edited

Loading

iarspider commented Jul 5, 2024 •

edited

Loading